Add fixed-size binary support by connortsui20 · Pull Request #8865 · vortex-data/vortex

connortsui20 · 2026-07-20T18:52:37Z

Stacked on #8864.

Summary

add the FixedSizeBinary logical dtype, scalar representation, canonical VTable, builder, and serialization support
reuse the shared fixed-width physical storage introduced by the base PR while keeping a distinct vortex.fixed_size_binary array ID
add Arrow, Python, FFI, CUDA, file, compute, fuzz, and conformance integration
keep FixedSizeList as a separate canonical representation

codspeed-hq · 2026-07-20T19:16:09Z

Merging this PR will regress 2 benchmarks

⚠️

Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 3 improved benchmarks
❌ 2 regressed benchmarks
✅ 1873 untouched benchmarks
⏩ 3 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	Simulation	`new_raw_prim_test_between[i32, 2048]`	33.8 µs	39 µs	-13.41%
❌	Simulation	`decompress[u64, (10000, 256)]`	55.4 µs	62.5 µs	-11.24%
⚡	Simulation	`copy_nullable[65536]`	1.4 ms	1 ms	+31.8%
⚡	Simulation	`decompress[u8, (1000, 4)]`	31.1 µs	24 µs	+29.44%
⚡	Simulation	`copy_non_nullable[65536]`	1,094.1 µs	913.8 µs	+19.73%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing ct/fixed-width-encoding (d36c772) with ct/fixed-width-primitive-decimal (80db289)}

3 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

github-actions · 2026-07-20T20:01:06Z

Polar Signals Profiling Results

Latest Run

Status	Commit	Job	Attempt	Link
🟢 Done	`d36c772`		1	Explore Profiling Data

Previous Runs (6)

Status	Commit	Attempt	Link
🟢 Done	`c924a06`	1	Explore Profiling Data
🟢 Done	`97c8e02`	1	Explore Profiling Data
🟢 Done	`0592a4f`	1	Explore Profiling Data
🟢 Done	`8dedc68`	1	Explore Profiling Data
🟢 Done	`8ef7bb2`	1	Explore Profiling Data
🟢 Done	`5dd799a`	1	Explore Profiling Data

Powered by Polar Signals Cloud

github-actions · 2026-07-20T20:02:00Z

Benchmarks: Vortex queries 📖

Verdict: No clear signal (low confidence)
Attributed Vortex impact: +3.0%
Engines: DataFusion No clear signal (+6.9%, low confidence) · DuckDB No clear signal (-0.6%, low confidence)
Vortex (geomean): 1.007x ➖
Parquet (geomean): 0.992x ➖
Shifts: Parquet (control) -0.8% · Median polish +0.8%

How to read Verdict and Engines

Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

datafusion / vortex-file-compressed (1.039x ➖, 0↑ 0↓)

name	PR `d36c772` (ns)	base `b7ee873` (ns)	ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed	9718414	9472638	1.03
vortex_q01/datafusion:vortex-file-compressed	6262842	5947344	1.05

datafusion / parquet (0.973x ➖, 0↑ 0↓)

name	PR `d36c772` (ns)	base `b7ee873` (ns)	ratio (PR/base)
vortex_q00/datafusion:parquet	19766481	20526611	0.96
vortex_q01/datafusion:parquet	4516565	4597130	0.98

duckdb / vortex-file-compressed (1.005x ➖, 0↑ 0↓)

name	PR `d36c772` (ns)	base `b7ee873` (ns)	ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed	9990981	10002024	1.00
vortex_q01/duckdb:vortex-file-compressed	5899398	5833329	1.01

duckdb / parquet (1.012x ➖, 0↑ 0↓)

name	PR `d36c772` (ns)	base `b7ee873` (ns)	ratio (PR/base)
vortex_q00/duckdb:parquet	23420289	23225002	1.01
vortex_q01/duckdb:parquet	9457504	9318548	1.01

File Size Changes (1 files changed, +0.0% overall, 1↑ 0↓)

File	Scale	Format	Base	HEAD	Change	%
test.vortex	1.0	vortex-file-compressed	80.59 MB	80.59 MB	+48 B	+0.0%

Totals:

vortex-file-compressed: 80.85 MB → 80.85 MB (+0.0%)

codecov · 2026-07-20T22:22:38Z

Codecov Report

❌ Patch coverage is 58.04133% with 467 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.00%. Comparing base (35a4157) to head (8dedc68).

Files with missing lines	Patch %	Lines
vortex-array/src/arrays/fixed_size_binary/mod.rs	60.16%	194 Missing ⚠️
encodings/sparse/src/canonical.rs	0.00%	44 Missing ⚠️
vortex-array/src/builders/fixed_size_binary.rs	55.55%	44 Missing ⚠️
...rtex-array/src/arrays/constant/vtable/canonical.rs	0.00%	17 Missing ⚠️
...x-array/src/scalar_fn/fns/binary/compare/nested.rs	0.00%	17 Missing ⚠️
vortex-array/src/arrays/dict/execute.rs	0.00%	14 Missing ⚠️
vortex-array/src/canonical.rs	80.28%	14 Missing ⚠️
vortex-datafusion/src/convert/scalars.rs	33.33%	14 Missing ⚠️
vortex-array/src/arrays/masked/execute.rs	0.00%	11 Missing ⚠️
vortex-arrow/src/convert.rs	0.00%	11 Missing ⚠️
... and 26 more

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

connortsui20 added the changelog/feature A new feature label Jul 20, 2026 — with ChatGPT Codex Connector

connortsui20 force-pushed the ct/fixed-width-encoding branch from dde9039 to e13e63c Compare July 20, 2026 19:26

connortsui20 force-pushed the ct/fixed-width-primitive-decimal branch from 3ba3236 to 76fd71a Compare July 20, 2026 19:26

connortsui20 force-pushed the ct/fixed-width-encoding branch from e13e63c to 5dd799a Compare July 20, 2026 19:50

connortsui20 force-pushed the ct/fixed-width-primitive-decimal branch from 76fd71a to f00395e Compare July 20, 2026 19:50

connortsui20 force-pushed the ct/fixed-width-primitive-decimal branch from f00395e to 35a4157 Compare July 20, 2026 21:35

connortsui20 force-pushed the ct/fixed-width-encoding branch from 5dd799a to 8dedc68 Compare July 20, 2026 21:35

connortsui20 force-pushed the ct/fixed-width-primitive-decimal branch from 35a4157 to 2e0b1e2 Compare July 20, 2026 22:05

connortsui20 force-pushed the ct/fixed-width-encoding branch from 8dedc68 to 8ef7bb2 Compare July 20, 2026 22:05

connortsui20 force-pushed the ct/fixed-width-primitive-decimal branch from 2e0b1e2 to 35a4157 Compare July 20, 2026 22:20

connortsui20 force-pushed the ct/fixed-width-encoding branch from 8ef7bb2 to 8dedc68 Compare July 20, 2026 22:20

connortsui20 force-pushed the ct/fixed-width-encoding branch from 8dedc68 to 0592a4f Compare July 20, 2026 22:33

connortsui20 force-pushed the ct/fixed-width-primitive-decimal branch from 35a4157 to 2880b5f Compare July 20, 2026 22:33

connortsui20 force-pushed the ct/fixed-width-encoding branch from 0592a4f to 97c8e02 Compare July 20, 2026 22:42

connortsui20 force-pushed the ct/fixed-width-primitive-decimal branch from 2880b5f to f124538 Compare July 20, 2026 22:42

connortsui20 force-pushed the ct/fixed-width-encoding branch from 97c8e02 to c924a06 Compare July 21, 2026 17:26

connortsui20 force-pushed the ct/fixed-width-primitive-decimal branch from f124538 to 8f8869b Compare July 21, 2026 17:26

connortsui20 added 3 commits July 21, 2026 13:27

Add fixed-size binary canonical type

9700197

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

Integrate fixed-size binary support

0a53354

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

Complete fixed-size binary language bindings

d36c772

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

connortsui20 force-pushed the ct/fixed-width-primitive-decimal branch from 8f8869b to 80db289 Compare July 21, 2026 17:30

connortsui20 force-pushed the ct/fixed-width-encoding branch from c924a06 to d36c772 Compare July 21, 2026 17:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add fixed-size binary support#8865

Add fixed-size binary support#8865
connortsui20 wants to merge 3 commits into
ct/fixed-width-primitive-decimalfrom
ct/fixed-width-encoding

connortsui20 commented Jul 20, 2026 •

edited

Loading

Uh oh!

codspeed-hq Bot commented Jul 20, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 20, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 20, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jul 20, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

connortsui20 commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

codspeed-hq Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will regress 2 benchmarks

Performance Changes

Footnotes

Uh oh!

github-actions Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Polar Signals Profiling Results

Latest Run

Uh oh!

github-actions Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks: Vortex queries 📖

Uh oh!

codecov Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

connortsui20 commented Jul 20, 2026 •

edited

Loading

codspeed-hq Bot commented Jul 20, 2026 •

edited

Loading

github-actions Bot commented Jul 20, 2026 •

edited

Loading

github-actions Bot commented Jul 20, 2026 •

edited

Loading

codecov Bot commented Jul 20, 2026 •

edited

Loading